On Markov chains with rewards

On sampling with Markov chains

Empirical Bayes Estimation in Nonstationary Markov chains

Estimation procedures for nonstationary Markov chains appear to be relatively sparse. This work introduces empirical  Bayes estimators  for the transition probability  matrix of a finite nonstationary  Markov chain. The data are assumed to be of  a panel study type in which each data set consists of a sequence of observations on N>=2 independent and identically dis...

Optimal Control of Ergodic Continuous-Time Markov Chains with Average Sample-Path Rewards

In this paper we study continuous-time Markov decision processes with the average sample-path reward (ASPR) criterion and possibly unbounded transition and reward rates. We propose conditions on the system’s primitive data for the existence of -ASPR-optimal (deterministic) stationary policies in a class of randomized Markov policies satisfying some additional continuity assumptions. The proof o...

On Triplet Markov Chains

The restoration of a hidden process X from an observed process Y is often performed in the framework of hidden Markov chains (HMC). HMC have been recently generalized to triplet Markov chains (TMC). In the TMC model one introduces a third random chain U and assumes that the triplet T = (X,U, Y ) is a Markov chain (MC). TMC generalize HMC but still enable the development of efficient Bayesian al...

On Markov Chains with Continuous State Space

In this expository paper, we p r o ve the following theorem, which m a y be of some use in studying Markov chain Monte Carlo methods like hit and run, the Metropolis algorithm, or the Gibbs sampler. Suppose a discrete-time Markov c hain is aperiodic, irreducible, and there is a stationary probability distribution. Then from almost all starting points the distribution of the chain at time n conv...